Skip to content

fix: VRAM and memory leaks in window, session, output, and multi-GPU paths - #2500

Open
minyek wants to merge 11 commits into
pop-os:masterfrom
minyek:vram-leak-fixes
Open

fix: VRAM and memory leaks in window, session, output, and multi-GPU paths#2500
minyek wants to merge 11 commits into
pop-os:masterfrom
minyek:vram-leak-fixes

Conversation

@minyek

@minyek minyek commented Jun 25, 2026

Copy link
Copy Markdown

Summary

Nine independent fixes to compositor-owned leaks. Each keeps GPU buffers or
compositor objects alive after the window, session, or output that needed them is
gone, so memory climbs during minimize, zoom, monitor-hotplug, multi-GPU, and
activation workflows and never returns to the idle baseline.

This is the cosmic-comp half of a two-part investigation into a steady VRAM climb
on the proprietary NVIDIA driver; the companion
smithay PR fixes library-level leaks in the
allocator and GLES renderer. These fixes are self-contained against current
master and need no smithay change. A residual VRAM climb on NVIDIA — memory not
reclaimed for sampled, cross-process dmabuf imports — reproduces with no
compositor code and is an NVIDIA driver issue, not a bug in either PR.

The NVIDIA driver issue has been reported here

What's fixed

  1. Minimized windows — a client that disconnects while minimized left its
    MinimizedWindow holding strong references, trapping GPU textures; now dropped
    via an IsAlive retain in Workspace::refresh / WorkspaceSet::refresh.
  2. Cursor image cacheimage_cache accumulated animation frames without
    bound across cursor-shape changes; now cleared when the shape changes.
  3. Stale Wayland activationspending_activations entries for Wayland
    surfaces that died before completing were never removed; now pruned in
    Shell::refresh.
  4. Stale X11 activations — the ActivationKey::X11 retain filter always
    returned true, so entries for X11 windows that died before mapping
    accumulated; now kept only while a matching pending window exists.
  5. Session-lock surfaces — removing an output left its LockSurface in
    SessionLock.surfaces; now removed on output teardown.
  6. Postprocess texturespostprocess_textures kept a DRM node's
    GlesTextures after the node was removed; now freed on GPU hot-unplug.
  7. Zoom reference cycleOutput → OutputZoomState → IcedElement → Output
    kept the output alive after disconnect; broken by calling output_leave() on
    the zoom element before removing the output.
  8. Main-thread renderer cleanup queues — the main renderers only draw on
    output (re)configuration, so they never drained their GL destruction queues,
    pinning imported client buffers in VRAM indefinitely; now drained periodically
    from refresh().
  9. Multi-GPU client disconnect — the disconnect handler broke out of the
    device loop after the first node, leaking the client's id (and its imported
    buffers) on every other device it used; now removed from all devices.

Testing

Both binaries were instrumented for this hunt — this compositor and its smithay
dependency — and validated with a SIGUSR1 resource census over a multi-day
dual-output NVIDIA session. Every targeted container stayed zero or bounded under
load: minimized windows, zoom states, pending activations, and session-lock
surfaces returned to zero, and the main-thread GL cleanup queues drained fully
(15.4M framebuffers cycled through and freed over three days). The only residual
is an NVIDIA driver-side VRAM pool that reproduces with no compositor code, not a
compositor leak. Compiles against the pinned smithay (rev 85f83ab).

AI assistance

This work was developed with AI assistance (Claude Code); use of AI-generated
code is disclosed in the commit messages per the contribution guidelines. All
changes have been reviewed and are understood by the author.

Checklist

  • I have disclosed use of any AI generated code in my commit messages.
  • I understand these changes in full and will be able to respond to review comments.
  • My change is accurately described in the commit message.
  • My contribution is tested and working as described.
  • I have read the Developer Certificate of Origin and certify my contribution under its conditions.

@jacobgkau
jacobgkau requested review from a team June 25, 2026 15:15
@jklgrasso jklgrasso self-assigned this Jun 25, 2026
jklgrasso
jklgrasso previously approved these changes Jun 26, 2026
@minyek

minyek commented Jul 16, 2026

Copy link
Copy Markdown
Author

Resolved the conflict with master (upstream's new KeyboardLayoutState::refresh landed at the same spot in refresh() as the KMS cache cleanup — kept both); the approval was auto-dismissed by the push, mind re-approving?

@Drakulix Drakulix left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks mostly good. Just one nitpick.

Comment thread src/backend/kms/mod.rs
}
self.last_renderer_cleanup = Instant::now();
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of adding a new periodic cleanup function, should we be able to simply do a cleanup (or schedule a cleanup with loop_handle.insert_idle after output (re-)configuration?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that'd fix the issue as the queue is populated at drop time, not import time — surfaces imported during (re-)configuration only become garbage later, when the window closes or the client exits. A cleanup right after configuration would run before that garbage exists and then never again, so e.g. a client exiting an hour after the last hotplug would stay pinned in VRAM until the next mode change. Since the resources are queued at arbitrary points (window close, client exit, cache eviction inside smithay), there's no single event that covers them all — hence the throttled drain (a make_current + empty try_iter at most every 2s).

We could make this event-based with a smithay change (e.g. a notifier/ping when a cleanup queue gets work), but for this PR I decided to keep the change contained to cosmic-comp.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. I think ideally we would simply have a method on the GraphicsApi/MultiRenderer and GlesRenderer to invalidate all caches. We know we don't need them and cleaning them up periodically still seems quite unnecessary to me.

Would you consider making a smithay PR to add such methods?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I'm happy to make the smithay change and I agree it's a better solution. I'm away on holiday, so when I'm back next week I'll complete my testing and submit the smithay PR. Once that's approved, I'll update this PR.

minyek added 11 commits August 15, 2026 14:29
A client that disconnected while minimized left its MinimizedWindow holding strong
references indefinitely, trapping the window's GPU textures. Add an IsAlive impl
for MinimizedWindow and retain only live entries in Workspace::refresh and
WorkspaceSet::refresh.

Developed with AI assistance (Claude Code); all changes have been reviewed and
are understood by the author.
The cursor image_cache accumulated animation frames without bound as it grew
across shape changes. Track the last cursor icon and clear the cache when the
shape changes.

Developed with AI assistance (Claude Code); all changes have been reviewed and
are understood by the author.
…aces

pending_activations entries for Wayland surfaces that died before completing were
never removed and accumulated indefinitely. Drop them during Shell::refresh once
the surface is no longer alive.

Developed with AI assistance (Claude Code); all changes have been reviewed and
are understood by the author.
Removing an output left its LockSurface in SessionLock.surfaces, retaining the
surface's GPU buffer. Remove the output's entry when the output is torn down.

Developed with AI assistance (Claude Code); all changes have been reviewed and
are understood by the author.
The retain filter for ActivationKey::X11 always returned true, so entries for X11
windows that died before completing mapping accumulated indefinitely. Keep an X11
entry only while a matching pending window still exists.

Developed with AI assistance (Claude Code); all changes have been reviewed and
are understood by the author.
postprocess_textures kept the node's GlesTexture render buffers after a DRM node
was removed, leaking them on GPU hot-unplug. Drop the node's entry in
node_removed.

Developed with AI assistance (Claude Code); all changes have been reviewed and
are understood by the author.
OutputZoomState, stored in Output.user_data(), holds an IcedElement whose outputs
set contains the same Output, so the Output never drops after disconnect — leaking
it, its user_data, and the GPU textures held by the IcedElement. Call
output_leave() on the zoom element before removing the output to break the cycle.

Developed with AI assistance (Claude Code); all changes have been reviewed and
are understood by the author.
A multi-GPU client can be active on several render nodes. The disconnect handler
broke out of the device loop after the first node it freed, leaking the client's
id (and its imported buffers) on every other device it had used. Remove the client
from every device, then refresh used devices once if any were freed.

Developed with AI assistance (Claude Code); all changes have been reviewed and
are understood by the author.
Moves onto the revision carrying `Renderer::invalidate_caches` and the
multi-GPU cache methods the next commit calls into. The bump crosses three
API changes:

- The tablet protocol handling moved from `wayland::tablet_manager` to
  `input::tablet` and became event-struct based, with an explicit
  `TabletSeatHandler::ToolFocus` and per-frame `frame()` calls. Axis state
  is now sent as one `AxisFrame` built from the libinput event's
  changed-axis flags, which is what the previous sequence of individual
  setter calls expressed.
- `PointerConstraintRef::deactivate` takes the state, surface and pointer,
  because smithay now calls `PointerConstraintsHandler::remove_constraint`
  from it; the handler also receives the constraint being removed.
- `remove_constraint` therefore runs with the per-surface constraint mutex
  held and, on the pointer-leave path, from inside pointer dispatch.
  Applying the cursor position hint there deadlocks the compositor, since
  `apply_cursor_hint` retakes that mutex and drives `pointer.motion`, so
  the seat lookup and the warp are deferred to an idle callback. The hint
  is now honoured when the constraint is deactivated as well as when it is
  destroyed, which is what `zwp_locked_pointer.set_cursor_position_hint`
  specifies; clearing the hint on the first makes the second a no-op.

The lockfile also moves `drm-ffi` to 0.9.1 and `drm-sys` to 0.8.1, which the
new revision requires, and drops `rand` with it.

Developed with AI assistance (Claude Code); all changes have been reviewed and
are understood by the author.
The renderers owned by the main thread's `GpuManager` - used for output
(re-)configuration, screenshots and image-copy captures - draw only on
demand, and a renderer only flushes its GL destruction queue and only
re-imports its caches when it draws. A dead client's textures therefore
stay queued, and a live client's imports stay cached, until whatever
happens to draw next, which may be minutes or hours away. Both pin the
buffers in VRAM for that whole time. The per-output render threads have
no such problem: they draw every frame.

Tie the two halves of the cleanup to the events that create the garbage
rather than to a timer:

- Destruction schedules a drain. `buffer_destroyed` and
  `CompositorHandler::destroyed` set a flag that `refresh` acts on with
  `GpuManager::cleanup_texture_cache`, so the queued deletions are flushed
  on the next event-loop pass instead of at the next draw. Batching through
  the flag keeps a client exit that destroys hundreds of surfaces to one
  drain.
- Infrequent draws invalidate afterwards. Output (re-)configuration,
  screenshots and image-copy captures call `invalidate_caches` once they
  are done, since the imports they just cached provide no benefit before
  the next such draw and the next one re-imports what it needs.

Both go through the `GpuManager` cache methods rather than a hand-rolled
loop over `devices_mut`. Such a loop cannot reach the buffers cached for
copying between a render and a target node: those belong to the manager,
keyed by node pair, rather than to any one renderer. `offscreen_renderer`
builds exactly such a render-to-target renderer for screencopy and
screenshots, so a per-device loop left a full output-sized dmabuf per node
pair pinned in VRAM.

Captures drain rather than invalidate, because a screencast session
captures repeatedly and its imports are worth keeping between frames.
Screenshots on the Glow path do neither: the winit and X11 backends redraw
every frame, and the KMS software fallback holds no VRAM.

The per-output render threads get the same treatment for consistency, which
supersedes `f86cd933`: their end-of-draw loop over `devices_mut` predates the
manager-level methods and is what those methods were added to replace.

Developed with AI assistance (Claude Code); all changes have been reviewed and
are understood by the author.
Removing a connector dropped its `Surface` without joining the thread it
owns. A bare drop only signals the thread to end. If a later reconfigure
takes the DRM compositor write lock before the detached thread finishes
its in-flight frame, the thread blocks on the read lock, never observes
`End`, and so never releases its renderer, swapchain and postprocess
offscreens. That strands a full set of output-sized render targets in
VRAM for the rest of the session: compositor VRAM does not return to its
previous floor after a display is unplugged, and each subsequent unplug
adds another set.

Join the thread so those resources are released before the connector's
removal completes. Unlike `apply_config_for_outputs`, this path holds no
compositor lock, so the join cannot deadlock against the thread's own
read lock; the two other `drop_and_join` call sites rely on the same
property.

Developed with AI assistance (Claude Code); all changes have been reviewed and
are understood by the author.
@minyek

minyek commented Aug 15, 2026

Copy link
Copy Markdown
Author

Smithay/smithay#2139 has landed, so this branch is rebased onto current master (f97a852e) and force-pushed. The 2 s drain timer you asked me to replace is gone; three new commits take its place. The other eight commits are as reviewed, apart from formatting-only rustfmt/clippy touch-ups in two.

1. chore: update smithay bumps the pin to 347b2b3 (the revision with Renderer::invalidate_caches and the GpuManager cache methods) and adapts to the API changes in between: the input::tablet rewrite, and the new pointer-constraints contract — remove_constraint now runs with the constraint mutex held inside pointer dispatch, where applying the cursor hint deadlocks, so the hint is applied from an idle callback, and on deactivation as well as destruction per the set_cursor_position_hint spec.

2. fix(kms): drain and invalidate the main-thread renderer caches is the event-driven cache management you asked for: buffer/surface destruction sets a flag that refresh drains via GpuManager::cleanup_texture_cache (one batched drain even when a client exit destroys hundreds of surfaces), and infrequent draws — output reconfiguration, screenshots, image-copy captures — call invalidate_caches when done (captures only drain, since a screencast reuses its imports every frame). Going through the GpuManager methods rather than a devices_mut loop matters: only the manager can reach the buffers cached per render/target node pair, which screencopy's offscreen_renderer creates — a per-device loop left a full output-sized dmabuf per pair pinned in VRAM. That is also why the loop f86cd933 added is dropped.

3. fix(kms): join the surface thread when its connector is removed is a one-liner: connector removal dropped the Surface without joining its thread, and a thread that missed the shutdown signal blocked on the compositor read lock forever, stranding its renderer, swapchain and postprocess offscreens — one full set of output-sized targets leaked per unplug.

Validation: a multi-day soak plus a scripted regression pass on an instrumented build with a GPU-resource census (27 censuses over 2.5 h, dual-output NVIDIA), covering output power-cycles, client churn, screencopy, workspace overview, and popups/zoom/minimize/cross-output moves. Every compositor-side counter returns to baseline or saturates flat, with no panic, GL error or DRM commit failure; the only residual growth is the known NVIDIA driver pooling, tracked separately with a standalone reproducer. One caveat: the cursor-hint deferral postdates the soak and is covered by manual testing of pointer-locking clients only.

@Drakulix

Copy link
Copy Markdown
Member

chore: update smithay bumps the pin to 347b2b3 (the revision with Renderer::invalidate_caches and the GpuManager cache methods) and adapts to the API changes in between: the input::tablet rewrite, and the new pointer-constraints contract — remove_constraint now runs with the constraint mutex held inside pointer dispatch, where applying the cursor hint deadlocks, so the hint is applied from an idle callback, and on deactivation as well as destruction per the set_cursor_position_hint spec.

We already have an update in progress that also implements tablet grabs and other features coming with the smithay update here: #2725

I'll ping you once that is merged and the smithay version used by cosmic-comp is new enough. I'd prefer to do that myself and then have you simply drop this commit and rebase the fixes on top of it. Thanks for validating all the fixes though!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants